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(54) Position detecting device for takeout apparatus 



(57) A position detecting device and a takeout ap- 
paratus capable of obtaining information on height or 
stage number of an object in astack of objects by atwo- 
dimensional image capturing device for an operation of 
taking out the object by a robot. A CCD camera (20) is 
arranged at a position above a place where workpieces 
(50) stacked in multiple stages are provided. An image 
processing device (30) determines a size of the image 
of the uppermost workpiece in a two-dimensional image 
captured by the camera. The height or stage number 
(A1 ..A4) of the workpieces is determined from the size 
of the workpiece image using a stored relation table or 



arithmetic expression representing relationship be- 
tween the height or stage number of the workpiece and 
the size of the workpiece. For a plurality of kinds of work- 
pieces, the relation table/arithmetic expression is stored 
for each kind of workpiece and selectively used in ac- 
cordance with a mark (M) on the workpiece which indi- 
cates the kind of workpiece. The obtained positional in- 
formation on height or stage number is sent to a robot 
controller (10) so that a robot (30) is operated to a 
takeout position which is corrected based on the posi- 
tional information for taking out the workpiece from a 
stack of workpieces. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present invention relates to a position de- 
tecting device suitable for use in handling a workpiece 
by a robot for an operation of taking out workpieces 
stacked in multiple stages or piling up workpieces suc- 
cessively, and takeout apparatus using the position de- 
tecting device. 

2. Description of Related Art 

[0002] In an operation of taking out workpieces 
stacked in multiple stages successively by a robot, the 
number of stages of the stacked workpieces and thus a 
position (particularly in a height direction) of the work- 
piece to be taken up change. In such a case, a sensor 
is used to recognize the position of the workpiece to be 
taken up. In this case, it has been deemed to be difficult 
to recognize the number of stages or height of the 
stacked workpieces based solely on information ob- 
tained by a two-dimensional image capturing device 
such as a camera. 

[0003] For this reason, the detection of the number of 
stages of the workpieces has been realized by methods 
such as one that uses a contact sensor for measuring 
height of an upper surface of the stacked workpieces, 
one that utilizes a so-called stereograph to detect the 
height of the workpieces based on images captured by 
two or more cameras, and one that uses a projection of 
a plurality of slit lights on the workpieces to obtain a 
three-dimensional image of the workpieces. 
[0004] However, use of the contact sensor for meas- 
uring height of the stacked workpieces has a problem 
of failing in contact with the upper surface of the work- 
pieces if the workpieces are displaced in a horizontal 
plane. Thus, in order to avoid this problem, it is neces- 
sary to use a two-dimensional visual sensor in combi- 
nation with the contact sensor. In addition, the method 
of the stereograph also has problems of need for instal- 
lation of a plurality of cameras and processing for es- 
tablish coordination between a plurality of images cap- 
tured by the plurality of cameras is required for realizing 
the detection of height of the stacked workpieces, and 
a difficulty in establishing the coordination. 

SUMMARY OF THE INVENTION 

[0005] An object of the present invention is to provide 
a position detecting device capable of detecting position 
of objects stacked in multiple stages by a simple ar- 
rangement using two-dimensional image capturing 
means without requiring arrangement and processing 
for the stereograph. Another object of the present inven- 
tion is to provide a takeout apparatus capable of per- 



forming an operation of taking out the object easily using 
the position detecting device. 

[0006] The present invention makes use of informa- 
tion on a size of an image of an object in a two-dimen- 
s sional image captured by the two-dimensional image 
capturing means to obtain positional information on 
height or stage number of the object. In addition, the 
present invention also provides an apparatus for per- 
forming an operation of taking out the object using the 
10 obtained positional information of the object. According 
to the present invention, the positional information on 
the height or stage number of the object can be obtained 
based on the two-dimensional image of the object, and 
therefore the information on the height or stage number 
'5 of an object in a stack of objects in multiple stages is 
obtained by a more simple arrangement than that in the 
conventional method, and the operation of taking out the 
object using the positional information is also realized. 
[0007] The position detecting device of the present in- 
vention comprises: image capturing means having an 
optical component for capturing a two-dimensional im- 
age including an image of an object; image processing 
means for determining a size of an image of the object 
in the two-dimensional image captured by the image 
capturing means; storing means for storing relational in- 
formation on a relationship between a distance from a 
reference position on an optical axis of the optical com- 
ponent of the image capturing means to the object on 
the optical axis, and the size of the image of the object 
in the two-dimensional image; and distance determining 
means for obtaining information on a distance from the 
reference position to the object in the direction of the 
optical axis based on the size of the image of the object 
determined by the image processing means and the re- 
lational information stored in the storing means. 
[0008] The reference position may be set in the image 
capturing means (for example at a position of a lens of 
a camera). 

[0009] The image capturing means may be arranged 
at a position above an uppermost one of workpieces of 
the same kind stacked in multiple stages on a loading 
plane, to capture a two-dimensional image including an 
image of the uppermost workpiece as the object. In this 
case, the reference position may be set to a position on 
the loading plane or a position of a workpiece at a pre- 
determined stage in the stacked workpieces. 
[0010] The distance determining means may obtain a 
distance between the reference position and the upper- 
most workpiece, or a stage number of the uppermost 
workpiece in the stacked workpieces calculated from 
the reference position. 

[0011] The storing means may store a relation table 
representing a relationship between the size of the im- 
age of the uppermost workpiece in the two-dimensional 
image and the stage number of the uppermost work- 
piece, and in this case, the distance determining means 
determines the stage number of the uppermost work- 
piece using the relation table. Alternatively, the storing 
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means may store an arithmetic expression representing 
a relationship between the size of the image of the up- 
permost workpiece in the two-dimensional image and 
the stage number of the uppermost workpiece, and in 
this case the distance determining means determines 
the stage number of the uppermost workpiece using the 
arithmetic expression. 

[001 2] The position detecting device of the present in- 
vention may further comprise input means for inputting 
information on a distance between the reference posi- 
tion and the uppermost workpiece, and storing means 
may store the relational information obtained based on 
two-dimensional images including images of the upper- 
most workpieces of different stack numbers captured by 
the image capturing means and the information on dis- 
tances between the reference position and the respec- 
tive uppermost workpieces inputted through the input 
means. 

[0013] The image capturing means may be mounted 
on a moving mechanism so that position of the image 
capturing means is changed by the moving mechanism. 
In this case, the moving mechanism is controlled such 
that the stacked workpieces is covered within a field of 
view of the image capturing means. 
[001 4] The image capturing means may capture two- 
dimensional images including images of uppermost 
workpieces of a plurality of stacks of workpieces placed 
at different positions, and the distance determining 
means may determine the highest stack of workpieces 
in the plurality of stacks of workpieces, and may deter- 
mine height or stage numbers of respective uppermost 
workpieces in the plurality of stacks of workpieces and 
thus the number of workpieces In respective stacks. 
[0015] The position detecting device of the present in- 
vention may further comprise setting means for setting 
information on kind of workpiece. In this case, the mov- 
ing mechanism may change the position of the image 
capturing means such that at least one of a plurality of 
stacks of workpieces of different kinds placed around 
the moving mechanism is covered within a filed of view 
of the image capturing means, and the distance deter- 
mining means may obtain the information on the dis- 
tance in accordance with the information on kind of 
workpiece set by the setting means. 
[001 6] The setting means may set the information on 
kind of workpiece according to a program command, an 
external signal or parameters, and alternatively, the set- 
ting means may set the information on kind of workpiece 
based on image processing of an image of the work- 
piece or image processing of a mark on the workpiece 
for indicating the kind of workpiece. 
[001 7] The position detecting device of the present in- 
vention may be combined with an object takeout means 
to constitute a takeout apparatus for taking out the ob- 
ject. In this case, the takeout apparatus may comprise 
takeout position determining means for determining a 
position for taking out the object by the object takeout 
device based on the information on the distance from 



4 

the reference position to the object in the direction of the 
optical axis determined by the distance determining 
means. 

[0018] The takeout apparatus may further comprise 
s two-dimensional position determining means for obtain- 
ing information on two-dimensional position and/or ori- 
entation of the object based on the image of the object 
in the two-dimensional image captured by the image 
capturing means. In this case, the takeout position de- 
10 termining means may determine the position for taking 
out the object based on a combination of the information 
on two-dimensional position and/or orientation of the ob- 
ject determined by the two-dimensional position deter- 
mining means and the information on the distance from 
f 5 the reference position to the object in the direction of the 
optical axis determined by the distance determining 
means. 

[0019] The takeout position determining means may 
correct the two-dimensional position of the object using 
20 the information on the data representing a distance from 
the reference position to the object in the direction of the 
optical axis, to thereby correct the position for taking out 
the object. 

[0020] The distance determining means may obtain 
25 height or the number of stages of objects stacked in mul- 
tiple stages to determine the number of objects, and in 
this case, the takeout apparatus may further comprise 
display means for displaying a notice or means for send- 
ing a signal to an external device when the number of 
30 objects reaches a predetermined value. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0021] 

35 

FIG. 1 is a schematic diagram of an overall config- 
uration of a workpiece takeout apparatus according 
to an embodiment of the present invention; 

40 FIG. 2 is a block diagram of main components of a 
robot controller used in the embodiment; 

FIG. 3 is a block diagram of main components of an 
image processing device used in the embodiment; 

45 

FIG . 4 is a diagram showing processing for convert- 
ing a two-dimensional position of an image of the 
workpiece into a two-dimensional position of the 
workpiece in a coordinate system set to a robot; 

50 

FIG. 5 is a schematic diagram of an overall config- 
uration of a workpiece takeout apparatus according 
to another embodiment of the present invention; 

55 FIG. 6 is a diagram showing a takeout operation for 
a plurality of stacks of workpieces; and 

FIG. 7 is a flowchart showing a takeout operation 
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for the plurality of stacks of workpieces. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0022] FIG. 1 shows an overall configuration of an ap- 
paratus for taking out workpieces one by one according 
to an embodiment of the present invention. A plurality 
of workpieces 50 (four workpieces in this example) are 
placed at a predetermined position to form a stack of 
workpieces in multiple stages. A CCD camera 20 for 
capturing a two-dimensional image of an object is ar- 
ranged above the stack of workpieces. In this embodi- 
ment, the CCD camera 20 is arranged at a fixed position 
and any mechanism for moving the CCD camera 20 is 
not provided. 

[0023] The CCD camera 20 constitutes a two-dimen- 
sional visual sensor in combination with an image 
processing device 30, and a two-dimensional image in- 
cluding an image of the workpieces 50 is captured by 
the CCD camera 20 in response to a command issued 
by the image processing device 30. The image process- 
ing device 30 detects an uppermost workpiece 50 in the 
stack of workpieces 50 in the two-dimensional image 
captured by the CCD camera 20 and obtains position/ 
orientation (e.g., two-dimensional position and orienta- 
tion on a plane) and size of the detected uppermost 
workpiece in the captured image. In addition, the image 
processing device determines a distance between the 
CCD camera 20 and the uppermost workpiece 50, or 
height or stage number of the uppermost workpiece 
(which indicates the uppermost workpiece is positioned 
at state from the bottom) using the detected size of the 
uppermost workpiece in the captured two-dimensional 
image. How to determine the above information will be 
described later. 

[0024] The image processing device 30 is connected 
to a robot controller 10 for controlling a robot 40. The 
data (including the information such as the two-dimen- 
sional position/orientation of the uppermost workpiece 
50, the distance between the uppermost workpiece 50 
and the CCD camera, the height or stage number of the 
uppermost workpiece) obtained by the image process- 
ing device 30 are sent to the robot controller 1 0. An op- 
eration program for holding the uppermost workpiece 50 
of the stacked workpieces 50 with a hand H is taught to 
the robot 40 in advance. There are a displacement be- 
tween the taught two-dimensional position/orientation 
and the actual two-dimensional position/orientation and 
a displacement between the taught height of the upper- 
most workpiece 50 and the actual height of the upper- 
most workpiece 50 in practice. 
[0025] The robot controller 10 performs a playback 
operation of the taught operation program to operate the 
robot 40 for taking out the workpieces 50 one by one. A 
target position in the operation of taking out the upper- 
most workpiece 50 is corrected on the basis of the data 
including the above information received from the image 



processing device 30. 

[0026] FIG. 2 is a block diagram showing main com- 
ponents of the robot controller 10. A well-known hard- 
ware architecture of a robot controller is applicable to 
5 the robot controller 10. Specifically, a main CPU 1, a 
memory 2, a teaching operation panel interface 3, a ser- 
vo control section 5, an external input/output interface 
6 and a communication interface 7 are connected to a 
bus line 8. 

w [0027] The memory 2 stores a system program for the 
main CPU 1 to generally control the entire system, op- 
eration programs taught for the operations of the robot 
40, taught data related to the taught operation programs 
and various parameters. 
15 [0028] A teaching operation panel 4 is connected to 
the teaching operation panel interface 3 to enable vari- 
ous manual operations such as a jog operation, starting 
of a playback operation and an emergency stop of the 
operation, as well as input and correction of various pro- 
grams and parameters. 

[0029] The servo control section 5 comprises servo 
control units 5al-5an of the number corresponding to the 
number "n" of servo-controlled axes. The servo control 
units 5al-5an drivingly control servo motors M1-Mn 
through servo amplifiers 5bl-5bn, respectively. The ex- 
ternal input/output interface 6 is provided for inputting 
signals from external equipments and outputting signals 
to the external equipments, and is used for inputting a 
signal indicating the kind of workpieces 50 from a work- 
piece feeding section, and for outputting a signal indi- 
cating that the stage number of the uppermost work- 
piece 50 reaches a predetermined value (of "1", for in- 
stance) to the workpiece feeding section. The determi- 
nation that the stage number of the uppermost work- 
piece 50 reaches the predetermined value is performed 
by the image processing device 30, and the determina- 
tion is sent to the robot controller 10 through the com- 
munication interface 7. The communication interface 7 
performs transfer of data to and from the image process- 
ing device 30 through a communication line, and in this 
embodiment, various information on the workpiece 50 
is sent from the image processing device 30 to the robot 
controller 10 via the communication line. . 
[0030] FIG. 3 shows main components of the image 
processing device 30. A hardware architecture of a well- 
known image processing device is applicable to the im- 
age processing device 30. Specifically, a CPU 31, a 
ROM 32, an image processor 33, an image capturing 
device interface 34, a manual data input device (MDI) 
35 with a display device, an image memory 36, a non- 
volatile memory 37, a RAM 38 and a communication in- 
terface 39 are connected to a bus line 40. 
[0031] In the ROM 32, a system program for the CPU 
31 to generally control the image processing device 30 
is stored. In the image memory 36, image data sent from 
the image capturing device through the interface 34 and 
image data such as gradation data obtained by image 
processing performed by the image processor 33 based 
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on the sent image data are stored. The manual data in- 
put device (MDI) 35 provides a man-machine interface 
for the image processing device 30 to allow manual op- 
erations such as input, editing and correction of the pro- 
gram data and the parameters. The RAM 38 is mainly 
used for temporary storage of data in arithmetic 
processing by the CPU 31 . The communication inter- 
face 39 is connected to the communication interface 7 
of the robot controller 1 0 via a communication line, and 
is used for transferring various data and signals be- 
tween the robot controller 1 0 and the image processing 
device 30. 

[0032] In the nonvolatile memory 37, data such as 
programs used for the image processing and analysis 
and parameters related to the programs are stored. 
These data includes information on a relationship be- 
tween a size of an image of a detected workpiece 50 in 
the captured image and a distance from a predeter- 
mined position on an optical axis of the CCD camera 20 
to the detected workpiece 50 in a direction of the optical 
axis, and a program and parameters for determining a 
distance from the predetermined position on the optical 
axis of the CCD camera 20 to the detected workpiece 
50 based on the above relationship and a size of the 
image of the detected workpiece 50 in the imaging 
plane. 

[0033] As described, the CCD camera 20 captures a 
two-dimensional image including an image of the work- 
piece 50 in response to a command issued from the im- 
age processing device 30. The captured two-dimen- 
sional image data are converted into gradation data by 
the image processing device 30 and stored in the image 
memory 36. Various conventional methods are applica- 
ble to determine position and size of the two-dimension- 
al image of the workpiece 50 in the captured image. 
[0034] For instance, there is a method in which the 
obtained image data are converted into binary image 
data composed of bright and dark pixel regions using a 
threshold value of an appropriate brightness, so that a 
shape of the image of the workpiece is determined as a 
bright or dark pixel region. The position of the image of 
the workpiece in the imaging frame is determined on the 
basis of a center of gravity of the bright or dark pixel 
region defining the shape of the image of the workpiece. 
The size of the image of the workpiece in the imaging 
frame is obtained on the basis of an area or a length of 
a principal axis of inertia of the pixel region defining the 
shape of the image of the workpiece. Alternatively, a 
template matching method by a so-called normalized 
correlation process may be also used. In this method, 
an image region of the workpiece to be detected is 
taught and stored as a template in advance, and the po- 
sition having the maximal correlation in pixel value be- 
tween the image captured by the camera and the tem- 
plate is obtained. 

[0035] In the template matching, there is also provid- 
ed a well-known method for using a plurality of tem- 
plates having different sizes provided by enlarging or re- 



ducing a taught template image to obtain a correlation 
in order to meet a change of the size of the image of the 
uppermost workpiece 50, to simultaneously calculate 
the position and the size of the image of the uppermost 

5 workpiece (such as the relative size of the detected 
workpiece on the basis of the size of a standard tem- 
plate). Thus, this method may be also applied to deter- 
mine the position and the size of the image of the up- 
permost workpiece according to the present invention. 

10 [0036] A method of obtaining data on height or stage 
number of the detected workpiece using the obtained 
information on the position and the size of the image of 
the uppermost workpiece in the imaging frame will be 
described. 

15 [0037] It is generally well known that projection by the 
camera for obtaining a two-dimensional image may be 
approximated to perspective projection by a pinhole 
camera. In the perspective projection, it is also well 
known that the size of an image projected onto an im- 
aging plane (e.g. , a CCD pixel plane of the CCD camera 
20 in this embodiment) is inversely proportional to a dis- 
tance Z between an object and a center of projection, i. 
e. a focus of a lens. 

[0038] Using the relationship between the distance Z 
and the size of the object, the distance Z between the 
uppermost workpiece and the CCD camera can be ob- 
tained based on the size of the image of the uppermost 
workpiece in the imaging frame. A length L (a length of 
the principal axis) of the image of the workpiece in a 
specific direction (a direction of the principal axis) or a 
length of one side of the image of the workpiece is ob- 
tained as an index representing the size of the image of 
the workpiece 50 by the image processing by the image 
processing device 30 in this embodiment. Other indexes 
such as an area of the region of the workpiece image 
may be used to represent the size of the workpiece im- 
age. 

[0039] First, in order to obtain the relationship be- 
tween the distance Z and the size of the workpiece 50, 
a two-dimensional image including an image of the 
workpieces with the uppermost workpiece 50 positioned 
in a different stage of first to n-th stage is captured. Then , 
the size of the image of the uppermost workpiece 50 in 
each of the captured two-dimensional images is deter- 
mined, to thereby obtain the size l(i) (i= 1 , 2, 3 ... n) of 
the image of the uppermost workpiece when the upper- 
most workpiece is positioned in the i-th stage. For ob- 
taining the size l(i) of the image of the workpiece at each 
stage, a single workpiece 50 may be placed in the i-th 
state so that an image of which is positioned at the cent- 
er of the imaging frame, or a plurality of workpieces may 
be placed in the i-th stage and an average of the sizes 
of all the images of the workpieces is used as the size I 
(i) of the image of the workpiece. Thus, based on the 
obtained data, a relation table indicating a relationship 
of the size L of the image of the uppermost workpiece 
50 and the stage number of of the uppermost workpiece 
50 corresponding to the distance Z is obtained. 
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[0040] Values in the relation table are stored in the 
nonvolatile memory 37 of the image processing device 
30. In actual detection, a two-dimensional image includ- 
ing an image of the uppermost workpiece 50 is captured 
and the size L of the image of the uppermost workpiece 
50 is determined. Then, referring to the relation table, 
the value of the size l(i) of the workpiece image in the 
relation table, which is most approximate to the value of 
the size L of the image of the detected workpiece is de- 
termined. Then, the value i corresponding to the size I 
(i) is read out from the relation table, to thereby deter- 
mine the stage number of the uppermost workpiece 50. 
[0041] The above processing for determining the 
stage number T of the uppermost workpiece is per- 
formed by the CPU 31 of the image processing device 
30 in this embodiment. Alternatively, the processing 
may be performed by the CPU 1 of the robot controller 
10, and in this case the size L of the workpiece image 
determined by the CPU 31 of the image processing de- 
vice 30 is sent to the robot controller 10 and the CPU1 
of the robot controller 10 determines the state number 
of the uppermost workpiece referring to the data in the 
relation table stored in the memory 2 of the robot 
controllerl 0. 

[0042] The term of "stage number" is typically used 
as stage number "1 " of the first stage, stage number "2" 
of the second stage, stage number "3" of the third stage 
(the level A3 in FIG. 1), stage number "4" of the fourth 
stage (the uppermost stage, i.e., the level A2 in FIG. 
1), which are counted upwardly from the loading 
plane (the level A4 in FIG. 1). Alternatively, other count- 
ing of the "stage number" may be also used. For in- 
stance, if the level A2 in FIG. 1 is set to a reference 
stage, which is a level of the upper surface of the upper- 
most workpiece of a new stack of the workpieces 50, 
stage number "0" is assigned to the workpiece having 
an upper surface at the level A2 and stage number "-1 " 
is assigned to the workpiece having an upper surface at 
the level A3. Similarly, when the level A3 is set to the 
reference stage, stage number "0" is assigned to the 
workpiece having an upper surface at the level A3 and 
stage number "+1 " is assigned to the workpiece having 
an upper surface at the level A2. 
[0043] Alternatively, an arithmetic expression repre- 
senting the relationship between the size L of the work- 
piece image and the height of the workpiece may be 
used to determine the stage number i or height h of the 
uppermost workpiece based on the detected size of the 
workpiece image. In an ideal camera establishing the 
central projection, the size (the length) of the workpiece 
image is inversely proportional to a distance between a 
focus of the camera and the workpiece. 
[0044] Thus, the relationship between the size L of the 
workpiece image and the stage number i of the work- 
piece can be expressed by an appropriate approximate 
expression using the known heights of the stacks of the 
workpieces of the respective stages. For instance, a lin- 
ear or quadratic or higher degree polynomial expression 



10 

may be used for the approximate expression of the re- 
lationship between the size L of the workpiece image 
and the workpiece height H. Coefficients of the approx- 
imate expression may be stored in the memory of the 
5 image processing device 30 or the robot controller 1 0 to 
be used for determining the workpiece height H based 
on the size L of the workpiece image in the captured 
two-dimensional image. The expression may be pre- 
pared for determining the stage number i of the upper- 
10 most workpiece instead of the workpiece height H. 
[0045] Since it is usual that a different kind of work- 
piece has a different thickness, the height of the upper- 
most workpiece varies depending on the kind of work- 
piece, and thus the relationship between the size of the 
15 uppermost workpiece image and the stage number or 
the height of the workpiece varies in dependence of the 
kind of workpiece 50. Thus, in a case of handling plural 
kinds of workpieces, the relation table or the relational 
expression is prepared for each kind of workpiece and 
stored in the memory of the image processing device 
30 or the robot controller 10. 

[0046] In an actual determination of the height/stage 
number of the workpiece, in order to decide which rela- 
tion table or arithmetic expression should be referred to , 
the image processing device 30 recognize a character- 
istic mark M made on the workpiece 50, for instance, to 
determine of the kind of workpiece, and obtain the infor- 
mation on the height/stage number of the workpiece re- 
ferring to the relational table/arithmetic expression pre- 
pared for the determined kind of workpiece. A barcode 
may be also used as the mark M, for instance. Alterna- 
tively, the mark M may be a label with printed figures 
(circle, triangle, square, etc. discriminable by the image 
processing) attached to the workpiece for the determi- 
nation of the kind of workpiece. 
[0047] Further, a plurality of kinds of programs for de- 
termining the height/stage number of the workpiece on 
the basis of the size of the workpiece image may be pre- 
pared and stored in the memory of the image processing 
device 30, and 

a program to be used for the kind of workpiece may be 
specified by a manual input or an external signal. Alter- 
natively, parameters for a common program may be 
specified by manual input or external signal. 
[0048] Referring to FIG. 4, description will be made 
on processing of converting the coordinate position of 
the image of the workpiece 50 in the imaging plane into 
a coordinate position in a coordinate system of the robot 
40 or an equivalent mechanism for taking out the work- 
piece. On a coordinate system for expressing the posi- 
tion of the workpiece image in the imaging plane, the 
size of the image capturing element (pixel) of the CCD 
camera may be used as a unit to express a distance. It 
is necessary to convert the coordinate position in the 
coordinate system set to the imaging plane into coordi- 
nate position in the robot coordinate system for taking 
out the workpiece 50 by the robot 40 using the detected 
position of the workpiece image. 



EP 1 324 268 A2 



25 



30 



35 



40 



45 



50 



6 



EP001324268 [file://C:\Documents and Settings\kflanaga\Desktop\EP001324268.cpc1 



Page 7 of 18 



11 EP 1 324 268 A2 12 



[0049] Data for the conversion between the coordi- 
nate systems is obtained by the processing of the well- 
known calibration. For instance, correspondence in di- 
rection and scale between the coordinate system set to 
the imaging plane and the robot coordinate system is 
established by recognizing images of two points, posi- 
tions of which are known in the robot coordinate system, 
in the image frame or by specifying the two points on 
the imaging frame by an operator. However, in the above 
calibration method, the relationship between an interval 
of pixels of the imaging plane and the scale of the robot 
coordinate system changes as the distance between the 
workpiece and the CCD camera is changed, so that an 
error between the coordinate position in the camera co- 
ordinate system and the coordinate position converted 
into the robot coordinate system is made large. 
[0050] As shown in FIG. 4, points on a line of sight 21 
passing through the focus of the CCD camera 20 form 
images at the same pixel position in the imaging plane 
according to the characteristics of perspective projec- 
tion. For instance, when the workpiece 50 is either at a 
position 51 or 52, these positions are on the same line 
of sight 21 , and therefore, may be recognized as the 
same position on screen. The more the positions 51 and 
52 are distant from each other, and the more the line of 
sight 2 1 is away from the center of the optical axis of the 
CCD camera (that is, the more the workpiece position 
is away from the center of the workpiece image), it will 
have a more effect of the perspective projection on the 
above conversion processing. Accordingly, some cor- 
rection is required in these cases. 
[0051] For correcting the error in the conversion, in- 
stance, the calibration is performed for each stage of the 
stacked workpieces and storing the conversion param- 
eters regarding the scales of the image coordinate sys- 
tem and the robot coordinate system for each stage of 
the stacked workpieces (the relationship between the 
direction in the image coordinate system and the direc- 
tion in the robot coordinate system remains unchanged 
and therefore only a change of the scale has to be con- 
sidered). Then, a value of the scale forthe stage number 
of the workpiece determined based on the workpiece 
image is used to in the conversion of the coordinate po- 
sition in the image coordinate system into the coordinate 
position in the robot coordinate system. 
[0052] With the above correction, a robot handling po- 
sition can be obtained accurately irrespective of the 
change of the workpiece height. In the case of handling 
plural kinds of workpieces, the relation tables or the 
arithmetic expressions representing the relationship be- 
tween the size of the workpiece image and the work- 
piece height are stored for the respective kinds of work- 
pieces, to cope with various kinds of workpieces by se- 
lecting the relation table or the arithmetic expression in 
accordance with the kind of the workpiece. 
[0053] The height H may be calculated on the basis 
of the loading plane A4 as shown in FIG. 1 , or otherwise, 
other reference positions may be also adopted in the 



same manner as that in the stage number. For instance, 
the level A2 (the uppermost level) or the level A3 or other 
level of a predetermined stage may be also set to a ref- 
erence height of "zero". Further, the level A1 of a lens 

5 plane of the CCD camera 20 may be also used as a 
reference position. In this case, height information cor- 
responds to "the distance between the CCD camera 20 
and the uppermost workpiece 50". 
[0054] In the above embodiment, the CCD camera 20 

10 is arranged in fixed position and any mechanism for 
moving the CCD camera is not provided. Alternatively, 
the CCD camera 20 may be arranged movably by 
mounting the camera on a moving mechanism. A robot 
may be typically used as the moving mechanism, and 

15 in this case, the robot may perform the operation of tak- 
ing out the workpiece, as shown in FIG. 5 according to 
another embodiment of the present invention. A work- 
piece takeout system of this embodiment is similar in 
arrangement to the system shown in FIG. 1 except that 

20 the CCD camera 20 in this embodiment is mounted on 
the robot 40, and therefore a repetitive description on 
the overall arrangement and individual components is 
omitted. 

[0055] A procedure for taking out workpieces one by 

25 one by the robot 40 as shown in FIG. 5 from three stacks 
1 , 2 and 3 of workpieces as shown in FIG. 6 will be de- 
scribed referring to a flowchart of FIG. 7. 1 n th is example, 
the kind of workpiece is determined by recognizing the 
mark M on the workpiece. Further, it is premised that the 

30 data representing the relationship between the stage 
number on the basis of the loading plane A4 and the 
height (the distance from the uppermost workpiece to 
the CCD camera) on the basis of the lens plane A1 are 
stored in the nonvolatile memory 37 of the image 

35 processing device 30 for each kind of workpiece in the 
form of the relation table or the arithmetic expression 
(the coefficient of the expression). Also, it is premised 
that positions F1 , F2 and F3 at which the camera cap- 
tures images of the stacks 1 , 2 and 3 of workpieces, re- 

40 spectively, and positions G1 , G2 and G3 for taking out 
the workpiece are taught in advance of the workpiece 
taking-up operation. It should be noted that the taught 
positions for taking out the workpiece are amended for 
use in the handling the workpiece. 

45 [0056] The steps in the procedure of the workpiece 
takeout operation will be summarized as follows. 
[0057] Step SO; The CPU 31 of the image processing 
device 30 initializes an index s (provided in the memory) 
indicating the stack number to "1 and sends the initial 

so value to the robot controller 1 0. 

[0058] Step S1 : The robot controller 1 0 operates the 
robot 40 to move to the detection position Fs for detect- 
ing of an s-th stack. 

[0059] Step S2; The image processing device 30 out- 
55 puts a command to the CCD camera 20 to capture an 
image. In response to the command, the CCD camera 
20 captures an image of the s-th stack of the workpieces 
and a signal representing the captured image is sent to 
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the image processing device 30. 
[0060] Step S3: The image processing device 30 de- 
termines position of an image of the uppermost work- 
piece. How to determine the position of the uppermost 
workpiece is already described. 
[0061] Step S4: The mark M is read and the image 
processing device 30 recognizes the kind of workpiece 
on the basis of the read mark. 

[0062] Step S5: The two-dimensional position/orien- 
tation of the workpiece are determined based on the po- 
sition/orientation of the workpiece image and the cali- 
bration data. 

[0063] Step S6: The size of the workpiece image is 
determined. How to determine the size of the workpiece 
image is already described. 

[0064] Step S7: Access to the relation table/arithmetic 
expression is made according to the determination re- 
sult in Step S4 to obtain the information on the height/ 
stage number of the detected workpiece based on the 
size of the workpiece image determined in Step S4. In 
this example, the height/stage number of the detected 
workpiece on the basis of the loading plane A4 is ob- 
tained. 

[0065] Step S8: The information on the height/stage 
number of the workpiece obtained in Step S7 is trans- 
ferred to the robot controller! 0 together with the data on 
the two-dimensional position/orientation of the work- 
piece obtained in Step S5. 

[0066] Step S9: The taught takeout position Gs is cor- 
rected using the information transferred in Step S8 and 
the robot 40 is operated to move to the corrected takeout 
position to perform the takeout operation (holding and 
carriage). 

[0067] Step S10: The image processing device 30 
waits for a notice of completion of the operation of taking 
out the uppermost workpiece by the robot 40 from the 
robot controller 10. 

[0068] Step S1 1 : It is determined whether or not the 
workpiece to be pucked up in Step S9 is a workpiece of 
the final stage based on the information on the height/ 
stage number of the workpiece obtained in Step S7. If 
a result of determination in Step S11 is YES, an alarm 
is displayed on the display device of the MDI 35 of the 
image processing device 30 (an alarm by sound may be 
also used) and the procedure proceeds to Step S12. If 
the result of determination in Step S11 is NO, the pro- 
cedure returns to Step S1 to repeatedly execute the 
processes of Steps S1 -S1 1 . 

[0069] Step S12: The CPU 31 of the image process- 
ing device 30 incrementally increases the index s indi- 
cating the stack number by "1". 
[0070] Step S13: The processing is terminated if the 
index s reaches the sum of the number of stacks (such 
as "3" in this example) and "1". If the index s does not 
exceed the number of stacks, the procedure returns to 
Step 1 to repeat the processes of Step S1 and subse- 
quent Steps for the next stack of workpieces. 
[0071] With the above procedure, the robot 40 can 



perform operations of taking out all the workpieces from 
the stacks 1 , 2 and 3 successively. Before starting the 
takeout operations, the detection of the height/stage 
number may be performed with respect to each of the 
5 stacks 1 to 3 as occasion demands, to obtain data on 
the height/stage number of each of the stacks 1 to 3, 
thereby determining the order of heights of the stacks 
to display such as "the highest stack", "the stack with 
the largest number of stage", etc. In addition, an alarm 
10 or an operation interrupt command may be also given 
when the height/stage number of each stack is deter- 
mined to be excessively high or large. 
[0072] According to the present invention, using the 
two-dimensional image capturing means, a distance be- 
15 tween the two-dimensional image capturing means and 
an uppermost workpiece, or the height or stage number 
of the workpiece can be detected, so that the positional 
information on the stack of workpieces in the height di- 
rection by a simple device in comparison with an existing 
device such as one that uses a contact sensor, one that 
uses a stereo observation and one that uses a projection 
of slit lights. In particular, the present invention provides 
simple means for obtaining the stage number of the up- 
permost workpiece of the stacked workpieces, whereby 
the position of the workpiece required for handling op- 
eration by the moving mechanism such as the robot is 
obtained. 

[0073] The position/orientation of the workpiece in a 
horizontal plane for taking out the workpiece by deter- 
mining the position and/or orientation of the workpiece 
image in the captured two-dimensional image. In addi- 
tion , the error (parallax) of the workpiece position on the 
horizontal plate depending on the height/stage number 
the workpiece is corrected to perform more precise de- 
tection of the two-dimensional position of the workpiece. 
[0074] By capturing a plurality of images including the 
workpiece images at different stages and inputting the 
information on the workpiece height (the distance be- 
tween the two-dimensional image capturing means and 
the workpiece, the height/stage number of the work- 
piece), the information on the relationship between the 
workpiece size and the workpiece height is easily ob- 
tained. The two-dimensional image capturing device 
may be mounted on the moving mechanism such as the 
robot, thereby allowing variations of the position/orien- 
tation of the two-dimensional image capturing device. 
[0075] The robot is operated such that the workpiece 
is covered in the field of view of the two-dimensional im- 
age capturing device for stable measurement. In the 
case where stacks of workpieces are placed in different 
positions, the highest one of the stacks of workpieces is 
determined and the takeout operation is started from the 
workpiece at the highest position, to thereby prevent an 
interference between the hand of the robot and the 
workpiece. To determine the number of workpieces in a 
stack or stacks is to determine the number of residual 
workpieces to be taken up by the robot hand, to thereby 
determine time for providing a new stack of workpieces. 
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[0076] When the number of residual workpieces 
reaches a predetermined value or less, an alarm may 
be issued to remind a provide of a new stack of work- 
pieces. By setting the information on a plurality of kinds 
of workpieces and selecting appropriate information in 
the set information, the position detection of the plurality 
of kinds of workpieces can be performed by the same 
detecting device. The selection of the appropriate infor- 
mation in accordance with the kind of workpiece may be 
performed automatically according to a program based 
on information on the kind of workpiece sent from an 
external controller. 

[0077] The kind of workpiece can be determined 
based on an image of a mark on the workpiece, to allow 
an automatic determination of the kind of workpiece. 
[0078] The present invention also provides a work- 
piece takeout apparatus, wherein the information on the 
height/stage number of the objects provided in multiple 
stages is obtained by the two-dimensional image cap- 
turing device and the obtained information is used for 
performing the takeout operation of the object. 



Claims 

1 . A position detecting device comprising: 

image capturing means having an optical com- 
ponent for capturing a two-dimensional image 
including an image of an object; 

image processing means for determining a size 
of an image of the object in the two-dimensional 
image captured by said image capturing 
means; 

storing means for storing relational information 
on a relationship between a distance from a ref- 
erence position on an optical axis of the optical 
component of said image capturing means to 
the object on the optical axis, and the size of 
the image of the object in the two-dimensional 
image; and 

distance determining means for obtaining infor- 
mation on a distance from the reference posi- 
tion to the object in the direction of the optical 
axis based on the size of the image of the object 
determined by said image processing means 
and said relational information stored in said 
storing means. 

2. A position detecting device according to claim 1 , 
wherein said reference position is set in said image 
capturing means. 

3. A position detecting device according to claim 1 , 
wherein said image capturing means is arranged at 
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a position above an uppermost one of workpieces 
of the same kind stacked in multiple stages on a 
loading plane, to capture a two-dimensional image 
including an image of the uppermost workpiece as 
s the object. 

4. A position detecting device according to claim 3, 
wherein said reference position is set to a position 
on said loading plane or a position of a workpiece 

10 at a predetermined stage in the stacked workpiec- 
es. 

5. A position detecting device according to claim 3, 
wherein said distance determining means obtains 

15 a distance between the reference position and the 
uppermost workpiece. 

6. A position detecting device according to claim 3, 
wherein said distance determining means obtains 
a stage number of the uppermost workpiece in the 
stacked workpieces calculated from the reference 
position. 

7. A position detecting device according to claim 6, 
wherein said storing means stores a relation table 
representing a relationship between the size of the 
image of the uppermost workpiece in the two-di- 
mensional image and the stage number of the up- 
permost workpiece, and said distance determining 
means determines the stage number of the upper- 
most workpiece using the relation table. 

8. A position detecting device according to claim 6, 
wherein said storing means stores an arithmetic ex- 
pression representing a relationship between the 
size of the image of the uppermost workpiece in the 
two-dimensional image and the stage number of the 
uppermost workpiece, and said distance determin- 
ing means determines the stage number of the up- 
permost workpiece using the arithmetic expression. 

9. A position detecting device according to claim 3, fur- 
ther comprising input means for inputting informa- 
tion on a distance between the reference position 
and the uppermost workpiece, wherein said storing 
means stores the relational information obtained 
based on two-dimensional images including imag- 
es of the uppermost workpieces of different stack 
numbers captured by said image capturing means 
and the information on distances between the ref- 
erence position and the respective uppermost 
workpieces inputted through said input means. 

10. A position detecting device according to claim 1, 
wherein said image capturing means is mounted on 
a moving mechanism so that position of said image 
capturing means is changed by said moving mech- 
anism. 
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11. A position detecting device according to claim 10, 
wherein said moving mechanism is controlled such 
that the stacked workpieces is covered within afield 
of view of said image capturing means. 

12. A position detecting device according to claim 3, 
wherein said image capturing means captures two- 
dimensional images including images of uppermost 
workpieces of a plurality of stacks of workpieces 
placed at different positions, and said distance de- 
termining means determines the highest stack of 
workpieces in the plurality of stacks of workpieces. 

13. A position detecting device according to claim 3, 
wherein said image capturing means captures two- 
dimensional images including images of uppermost 
workpieces of a plurality of stacks of workpieces 
placed at different positions, and said distance de- 
termining means determines height or stage num- 
bers of respective uppermost workpieces in the plu- 
rality of stacks of workpieces and thus the number 
of workpieces in respective stacks. 

14. A position detecting device according to claim 10, 
further comprising setting means for setting infor- 
mation on kind of workpiece, wherein said moving 
mechanism changes the position of said image cap- 
turing means such that at least one of a plurality of 
stacks of workpieces of different kinds placed 
around said moving mechanism is covered within a 
filed of view of said image capturing means, and 
said distance determining means obtains the infor- 
mation on the distance in accordance with the infor- 
mation on kind of workpiece set by said setting 
means. 

15. A position detecting device according to claim 14, 
wherein said setting means sets the information on 
kind of workpiece according to a program com- 
mand, an external signal or parameters. 

16. A position detecting device according to claim 14, 
wherein said setting means sets the information on 
kind of workpiece based on image processing of an 
image of the workpiece or image processing of a 
mark on the workpiece for indicating the kind of 
workpiece. 

17. A takeout apparatus comprising: 

object takeout means; 

a position detecting device including: image 
capturing means having an optical component 
for capturing a two-dimensional image includ- 
ing an image of an object; image processing 
means for determining a size of an image of the 
object in the two-dimensional image captured 
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by said image capturing means; storing means 
for storing relational information on a relation- 
ship between a distance from a reference po- 
sition on an optical axis of the optical compo- 
5 nent of said image capturing means to the ob- 

ject on the optical axis, and the size of the im- 
age of the object in the two-dimensional image; 
and distance determining means for obtaining 
information on a distance from the reference 
position to the object in the direction of the op- 
tical axis based on the size of the image of the 
object determined by said image processing 
means and said relational information stored in 
said storing means; and 

takeout position determining means for deter- 
mining a position for taking out the object by 
said object takeout means based on the infor- 
mation on the distance from the reference po- 
sition to the object in the direction of the optical 
axis determined by said distance determining 
means. 

18. A takeout apparatus according to claim 1 7, further 
comprising two-dimensional position determining 
means for obtaining information on two-dimension- 
al position and/or orientation of the object based on 
the image of the object in the two-dimensional im- 
age captured by said image capturing means, 
wherein said takeout position determining means 
determines the position for taking out the object 
based on a combination of the information on two- 
dimensional position and/or orientation of the object 
determined by said two-dimensional position deter- 
mining means and the information on the distance 
from the reference position to the object in the di- 
rection of the optical axis determined by said dis- 
tance determining means. 

19. A takeout apparatus according to claim 1 8, wherein 
said takeout position determining means corrects 
the two-dimensional position of the object using the 
information on the data representing a distance 
from the reference position to the object in the di- 
rection of the optical axis, to thereby correct the po- 
sition for taking out the object. 

20. A takeout apparatus according to claim 1 7, wherein 
said distance determining means obtains height or 
the number of stages of objects stacked in multiple 
stages to determine the number of objects, and said 
takeout apparatus further comprises display means 
for displaying a notice when the number of objects 
reaches a predetermined value. 

21 . A takeout apparatus according to claim 1 7, wherein 
said distance determining means obtains height or 
the number of stages of objects stacked in multiple 
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stages to determine the number of objects, and said 
takeout apparatus further comprises means for 
sending a signal to an external device when the 
number of objects reaches a predetermined value. 
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